Papers with self-supervised framework
AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decomposition (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown promise in multi-step reasoning tasks, yet relying on extensive manual labeling to provide procedural feedback remains a significant impediment. |
| Approach: | They propose a self-supervised framework that decomposes complex problems into manageable subquestions with a controllable granularity switch and sequentially applies reinforcement learning to iteratively improve the subquest solver. |
| Outcome: | The proposed framework improves performance on mathematical and commonsense reasoning tasks over SOTA. |
ScanEZ: Integrating Cognitive Models with Self-Supervised Learning for Spatiotemporal Scanpath Prediction (2025.acl-short)
Copied to clipboard
| Challenge: | ScanEZ framework provides a framework for predicting scanpaths during reading . masked modeling of eye movements and cognitive model simulations are used to kick-start training. |
| Approach: | They propose a framework for self-supervised learning that models scanpaths using synthetic data and a 3-D gaze objective inspired bymasked language modeling. |
| Outcome: | The proposed framework achieves state-of-the-art results on established datasets and is portable across different conditions. |
QA Domain Adaptation using Hidden Space Augmentation and Self-Supervised Contrastive Adaptation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Question answering models often suffer from performance deterioration upon deployment . |
| Approach: | They propose a self-supervised framework called QADA for QA domain adaptation . they propose to augment training QA samples with hidden space augmentation . |
| Outcome: | The proposed framework improves on multiple target datasets over state-of-the-art methods. |
Improving Event Causality Identification via Self-Supervised Representation Learning on External Causal Statement (2021.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for event causality identification (ECI) rely on labeled data, but the scale of annotated datasets is limited. |
| Approach: | They propose a self-supervised framework to learn context-specific causal patterns from external causal statements and adopt a contrastive transfer strategy to incorporate the learned context- specific causal patterns into the target ECI model. |
| Outcome: | The proposed method significantly outperforms existing methods on EventSto-ryLine and Causal-TimeBank (+2.0 and +3.4 points on F1 value respectively). |
Attention-based Contrastive Learning for Winograd Schemas (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Existing approaches to learn discriminative features using contrastive objective are lacking. |
| Approach: | They propose a self-supervised framework that leverages a contrastive loss directly at the level of self-attention. |
| Outcome: | The proposed framework outperforms all comparable unsupervised approaches while occasionally surpassing supervised ones. |
SelfORE: Self-supervised Relational Feature Learning for Open Relation Extraction (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for relation extraction use heuristics or distant-supervised annotations, but distant supervised methods make strong assumptions on entity cooccurrence without sufficient contexts. |
| Approach: | They propose a framework that exploits weak, self-supervised signals by leveraging large pretrained language models for adaptive clustering on contextualized relational features. |
| Outcome: | The proposed framework exploits weak, self-supervised signals on open-domain Relation Extraction . it bootstraps the self-supervised signals by improving contextualized features in relation classification . |
RSVP: Customer Intent Detection via Agent Response Contrastive and Generative Pre-Training (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing intent detection approaches have relied on adaptively pre-training language models with large-scale datasets, yet the predominant cost of data collection may hinder their superiority. |
| Approach: | They propose a self-supervised framework dedicated to task-oriented dialogues which incorporates agent responses for pre-training in a two-stage manner. |
| Outcome: | The proposed framework outperforms the state-of-the-art frameworks for task-oriented dialogues on two real-world customer service datasets. |
KBAlign: Efficient Self Adaptation on Specific Textual Knowledge Bases (2025.findings-emnlp)
Copied to clipboard
Zheni Zeng, Yuxuan Chen, Shi Yu, Ruobing Wang, Yukun Yan, Zhenghao Liu, Shuo Wang, Xu Han, Zhiyuan Liu, Maosong Sun
| Challenge: | Existing methods for retrieval-augmented generation (RAG) are limited and fine-tuning incurs prohibitive costs of external signals. |
| Approach: | They propose a self-supervised framework that enhances RAG systems through efficient model adaptation. |
| Outcome: | The proposed framework achieves 90% of the performance gain obtained through GPT-4-supervised adaptation while relying entirely on self-annotation of much smaller models. |
DNA: Denoised Neighborhood Aggregation for Fine-grained Category Discovery (2023.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods to learn compact cluster representations from coarsely labeled data are noisy and degrade the quality of learning. |
| Approach: | They propose a framework that encodes semantic structures of data into the embedding space . they retrieve k-nearest neighbors of a query as positive keys to capture similarities . |
| Outcome: | The proposed framework can retrieve more accurate neighbors and outperform state-of-the-art models by a large margin. |
MaskTab: Scalable Masked Tabular Pretraining with Scaling Laws and Distillation for Industrial Classification (2026.findings-acl)
Copied to clipboard
| Challenge: | Tabular data is high-dimensional, riddled with missing entries, and rarely labeled at scale. |
| Approach: | They propose a unified pre-training framework for industrial-scale tabular data . MaskTab encodes missing values via dedicated learnable tokens . |
| Outcome: | The proposed framework outperforms XGBoost and MaskTab-L on industrial-scale . it achieves +5.04% AUC and +8.28% KS over prior art under rigorous scaling . |